I had problems piping out the GUI install wizard with the DB2 install (though it didn't do what your server did!) even though the xserver worked fine for the other parts of the install so I used a response file which worked like a charm.
Did you run all the pre-reqs before doing the install?
Disable selinux
and the stuff in the below URL to the DB2 Infocenter:
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=/com.ibm.db2.luw.qb.server.doc/doc/r0008865.html